提示和背景

Prompt and Context

当你构建人工智能应用程序时，通常情况下，你会通过 API 调用托管的基础模型。这个 API 可能是基础模型供应商提供的，也可能是一个允许你轻松切换供应商的框架。

When you build AI applications, you typically invoke hosted foundational models through an API. This might be the API provided by the vendor of the foundational model, or it might be a framework that allows you to easily switch between providers.

您可以通过向基础模型发送提示并接收响应来调用它。您无疑很熟悉如何使用基础模型的 Web 用户界面来完成此操作。例如，在ChatGPT上，您可以在文本框中输入如下提示：

You invoke a foundational model by sending it a prompt and getting back a response. You are, doubtless, familiar with doing this by using the web user interface of a foundational model. For example, on ChatGPT, you might type a prompt like this one into the text box:

用德加的风格创作一幅铅笔素描，描绘一家四口在玩棋盘游戏。

Create a pencil sketch in the style of Degas depicting a family of four playing a board game

最简单的提示通常包含向模型发出指令，要求其执行某些内容生成任务。在这种情况下，模型会执行指令并返回包含所请求类型图像的响应（参见^图1-1）。 提示和回复都可以是多模态的——它们可以是文本，也可以是图像、视频或音频。

The simplest prompt typically consists of an instruction to the model that asks it to perform some content-generation task. In this case, the model follows the instruction and sends back a response that contains an image of the type requested (see Figure 1-1).¹ Both prompts and responses can be multimodal—they could be text, but they could also be images, video, or audio.

图 1-1.向基础模型发送提示以获取响应

更复杂的提示可能包括不仅是指令，还有上下文。这可能是模型在生成内容时应该使用的信息，也可能是要求模型扮演特定角色的信息。例如，我们可以告诉 Claude（在Claude.ai上）赋予某种个性，并在生成内容时使用一些信息：

A more complex prompt might consist not just of an instruction but also context. This might be information that the model should use in the generation, or it might involve asking the model to take on a specific role. For example, we could tell Claude (on Claude.ai) to take on a personality and use some information when generating content:

您是一位经验丰富的营销专家，非常熟悉德国大学城的图书市场。

《水之盟约》是一部小说，讲述了喀拉拉邦一个东正教圣托马斯基督教家庭三代人的故事。

写一段文字，向哥廷根书店的读者介绍这本书，并阐述它与当地的联系。

You are an expert marketer who is very familiar with the book market in university towns in Germany.

Covenant of Water is a novel that tells the story of three generations of an Orthodox Saint Thomas Christian family in Kerala.

Write a one-paragraph blurb introducing the book to readers at a bookstore in Göttingen, drawing local connections.

克劳德·索内特的回答考虑到了角色、背景和指示：

The response from Claude Sonnet takes into account the role, context, and instruction:

水之契约——喀拉拉邦文学之旅

在哥廷根历史悠久的城墙之内，学术传统与文学探索在此交融，我们隆重推出亚伯拉罕·维吉斯的鸿篇巨制《水之盟约》。这部气势恢宏的史诗巨著讲述了印度喀拉拉邦一个东正教圣托马斯基督徒家庭三代人的故事——喀拉拉邦复杂的宗教遗产或许能引起我们大学著名神学研究项目学生的共鸣。故事跨越数十年，在喀拉拉邦郁郁葱葱的内陆水域中徐徐展开。熟悉哥廷根与水之间渊源的读者——从神圣的甘塞利塞尔喷泉到蜿蜒的莱纳河——将会发现地理环境如何塑造命运，两者之间存在着惊人的相似之处。维尔吉斯编织了一个关于家庭秘密、医学谜团和代际纽带的故事，超越了文化界限，为我们国际学者和终身学习者群体提供了一次深刻的探索，探讨了水——无论是喀拉拉邦的季风还是下萨克森州的温柔雨水——如何成为一种隐喻和契约，将我们跨越时间和空间联系起来。

Covenant of Water - A Literary Journey to Kerala

Nestled between the historic walls of Göttingen, where academic tradition meets literary curiosity, we present Abraham Verghese’s masterful epic, “The Covenant of Water.” This sweeping saga follows three generations of an Orthodox Saint Thomas Christian family in Kerala, India—a region whose complex religious heritage might resonate with students of our university’s renowned theological studies program. As the story unfolds across decades amid Kerala’s lush backwaters, readers familiar with Göttingen’s own relationship with water—from the sacred Gänseliesel fountain to the meandering Leine River—will find surprising parallels in how geography shapes destiny. Verghese weaves a tale of family secrets, medical mysteries, and generational bonds that transcends cultural boundaries, offering our community of international scholars and lifelong learners a profound exploration of how water—whether in Kerala’s monsoons or Lower Saxony’s gentle rains—becomes both metaphor and covenant, connecting us across time and space.

使用模型提供商的 API

Using the Model Provider’s API

旨在实现以下功能的 API调用基础模型的方法在多种语言中都有应用——Python、Go、Java 和 TypeScript 都是常用的语言。为简单起见，本书将重点介绍 Python API。

APIs that are designed to invoke foundational models are available in a wide variety of languages—Python, Go, Java, and TypeScript are all commonly employed. For simplicity, we focus on Python APIs in this book.

引用克劳德十四行诗3要在 Python 中使用 Anthropic API，您可以按如下方式使用 Anthropic Python 包（完整代码在GitHub 上）：

To invoke Claude Sonnet 3 through the Anthropic API in Python, you could use the Anthropic Python package as follows (the full code is on GitHub):

import anthropic
client = anthropic.Anthropic(
    # defaults to os.environ.get("ANTHROPIC_API_KEY")
    api_key="YOUR_ANTHROPIC_API_KEY",
)
 
completion = client.messages.create(
    model="claude-3-7-sonnet-latest",
    system="You are an expert Python programmer.",
    messages=[
        {
            "role": "user",
            "content": [
               {
                  "type": "text",
                  "text": """
Write code to find the median value of a list of integers.
"""
               }
            ]
        }
    ]
)
 
print(completion.content[0].text)

在此 API 调用中，提示符有系统提示分为两部分：系统提示和用户提示。系统提示由开发者设置，用于指导模型的整体行为；而用户提示则更具动态性，为模型执行的特定任务提供具体指令。在本例中，AI 助手的角色已在系统提示中设置，而用户提示和上下文则以消息的形式发送。

In this API call, the prompt has been broken into two parts—a system prompt and a user prompt. The system prompt is set by the developer and guides the model’s overall behavior, while the user prompt is more dynamic and provides specific instructions for a specific task you want the model to perform. Here, the AI assistant’s role has been set in the system prompt while the user prompt and context are sent as messages.

使用与LLM无关的框架

Using an LLM-Agnostic Framework

要使用以下方式执行相同的任务如果使用 PydanticAI 框架，您可以使用如下代码（假设所需的 API 密钥已在环境变量中设置）：

To perform the same task using the PydanticAI framework, you’d use code such as the following (assuming the needed API key is set in an environment variable):

from pydantic_ai import Agent
agent = Agent('anthropic:claude-3-7-sonnet-latest',
              system_prompt="You are an expert Python programmer.")
 
result = agent.run_sync(
           "Write code to find the median value of a list of integers.")
print(result.data)

这样做的好处是，您可以通过将模型字符串更改为 `<model_name>` openai:gpt-4o-mini、google-vertex:gemini-2.0-flash`<model_name> groq:llama3-70b-8192`、`<model_name>` 等，轻松地在基础模型提供程序之间切换。（有关支持的模型的完整列表，请参阅Pydantic 的文档）。

The advantage here is that you can easily switch between foundational model providers by switching the model string to openai:gpt-4o-mini, google-vertex:gemini-2.0-flash, groq:llama3-70b-8192, and so on (see Pydantic’s documentation for the full list of models supported).

Pydantic API 中的类调用 Claude 模型被称为Agent。我们将在下一节讨论什么是代理，但在此之前，让我们结束关于调用基础模型的方法的讨论。

The class in the Pydantic API that invokes the Claude model is called Agent. We’ll discuss what agents are in the next section, but before that, let’s conclude our discussion of ways to invoke foundational models.

在本地运行模型

Running Your Model Locally

要在……上运行像 Llama 3 这样的模型在您的本地硬件上，您可以使用Ollama客户端下载并运行您想要使用的模型：

To run a model such as Llama 3 on your local hardware, you could use the Ollama client to download and run the model that you want to use:

ollama 跑 llama3.2

Ollama 通过 OpenAI API 公开了开放权重模型，因此您可以使用它：

Ollama exposes open-weights models with the OpenAI API, so you could use this:

from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
 
model = OpenAIModel(
    model_name='llama3.2', 
    provider=OpenAIProvider(base_url='http://localhost:11434/v1')
)

基础模型是如何创建的

How Foundational Models Are Created

与传统的机器学习应用不同，你的AI应用很少会包含训练阶段。相反，它们将基于预训练好的通用基础模型构建，这些模型能够执行各种各样的任务。在大多数情况下，你可以忽略基础模型的内部细节——我们在本书中包含这一部分只是为了帮助你理解相关的术语。

Unlike traditional machine learning applications, your AI applications will rarely include a training phase. Instead, you’ll build them on top of general-purpose foundational models that have been pretrained to perform a wide variety of tasks. You can, for the most part, ignore the internal details of the foundational model—and we’re including this section in this book only so you can understand the associated vocabulary.

截至2025年春季撰写本文时，DeepSeek是基础模型中训练方案信息最全面的模型。我们将利用这些信息来探讨创建基础模型的关键步骤（见图1-2）。虽然OpenAI、Google、Anthropic和Meta在创建GPT、Gemini、Claude和Llama时可能并未完全遵循这一流程，但它们的方法大体上可能类似。

At the time of writing in spring 2025, DeepSeek is the foundational model with the most available information on its training regimen. We’ll use that information to discuss the key steps (see Figure 1-2) involved in creating a foundational model. While OpenAI, Google, Anthropic, and Meta may not have followed this exact process in creating GPT, Gemini, Claude, and Llama, their methods are probably broadly similar.

DeepSeek 基础 LLM 是在包含 14.8 万亿个词元的多样化高质量语料库上进行预训练（如图 1-2的步骤 1 所示）。（作为对比，莎士比亚的作品大约包含 120 万个词元——因此 DeepSeek 的训练数据集相当于 1200 万部莎士比亚作品！）与早期基于词元训练的语言学习模型不同，现代语言学习模型是基于词元训练的。基于词元（即短字符序列），这类语言学习模型能够学习语言词汇表中不存在的内容，例如专有名词。除了数据量之外，DeepSeek 团队认为模型的高质量还得益于精心的数据整理，包括严格的去重流程。

The DeepSeek base LLM was pretrained (in Step 1 of Figure 1-2) on a diverse, high-quality corpus of 14.8 trillion tokens. (The works of Shakespeare, as a point of comparison, amount to about 1.2 million tokens—so the DeepSeek training dataset is equivalent to 12 million Shakespeares!) Unlike early LLMs, which were trained on words, modern LLMs are trained on tokens, which are short sequences of characters. This allows such LLMs to learn things that are not in the vocabulary of the language, like proper names. It isn’t just size that helps—the DeepSeek team attributes the high quality of their models to careful data curation, including rigorous deduplication processes.

图 1-2. DeepSeek-R1 模型训练的各个阶段

预训练阶段（图 1-2中的步骤 1 ）包括利用这个庞大的词元数据集进行训练，以提升模型的通用语言理解能力。此阶段的关键目标是训练模型，使其能够根据训练输入中先前词元构成的上下文预测下一个词元。这就是为什么你经常会听到人们称 LLM为下一个标记预测器——但下一个标记预测只是训练方案的第一阶段。

The pretraining stage (Step 1 in Figure 1-2) involves training on this massive dataset of tokens to develop the model’s general language-understanding capabilities. The key goal at this stage is to train the model to predict the next token, given a context consisting of the previous tokens in the training input. This is why you’ll often hear people call LLMs next-token predictors—but next-token prediction is only the first stage of the training regimen.

经过预训练后，模型该模型会经过监督式微调（SFT，图 1-2中的步骤 2 ），以提高其遵循指令和生成高质量答案的能力。此阶段使用精心整理的人工编写示例数据集。例如，Cohere 公司曾表示，他们会聘请执业医师、金融分析师和会计师来改进其模型。据推测，这些专业人士会在给定提示的情况下撰写理想的答案。最终成果DeepSeek-V3是一个混合专家（MoE）模型：这种优化方法允许模型拥有大量参数，但每个词元仅激活其中的一部分。DeepSeek-V3总共有6710亿个参数，但每个词元仅激活370亿个参数。这使得模型能够针对不同类型的指令使用不同的路径。

Following pretraining, the model undergoes supervised fine-tuning (SFT, Step 2 in Figure 1-2) to improve its ability to follow instructions and generate high-quality responses. This stage uses carefully curated datasets of human-written examples. Cohere, for example, has said it uses licensed physicians, financial analysts, and accountants to improve its models. Presumably, these practitioners write ideal answers given a prompt. The result, DeepSeek-V3, is a mixture of experts (MoE) model: an optimization that allows models to have a large number of parameters while activating only a fraction of them for each token. DeepSeek-V3 has 671 billion parameters in total, but only 37B are activated per token. This allows the model to use different pathways for different types of instructions.

强化学习阶段（图 1-2中的步骤 3 ）是这些模型会根据人类偏好进行进一步优化。这一步骤涉及带有人类反馈的强化学习（RLHF），这意味着向人类评分者展示一对生成的输出结果，并询问他们更喜欢哪一个。这种偏好调整有助于使模型的输出与人类的期望和价值观保持一致。这一阶段有时也称为偏好优化。

The reinforcement learning stage (Step 3 in Figure 1-2) is where the models are further refined based on human preferences. This step involves reinforcement learning with human feedback (RLHF), which means showing human raters a pair of generated outputs and asking them which one they prefer. Such preference tuning helps align the models’ outputs with human expectations and values. This stage is also sometimes called preference optimization.

在创建了包含少量人工编写示例和人工偏好的 DeepSeek-V3 之后，完整的 DeepSeek-R1 模型是通过以下多阶段过程开发的：

Once DeepSeek-V3 was created with a small number of human-written examples and human preferences, the full DeepSeek-R1 model was developed through the following multistage process:

1. 冷启动过程包括使用数千个冷启动数据点对基础模型（DeepSeek-V3-Base）进行微调，以奠定基础。

2. The cold start involved fine-tuning a base model (DeepSeek-V3-Base) with thousands of cold-start data points to lay a foundation.

3. 纯强化学习（RL）是指应用纯强化学习方法来增强推理能力。

4. Pure reinforcement learning (RL) involved applying a pure reinforcement learning approach to enhance reasoning skills.

5. 拒绝采样是指从上次成功的强化学习运行期间创建的数据分布中选择最佳样本。

6. Rejection sampling involved choosing the best examples from a distribution of data created during the last successful RL run.

7. SFT 涉及将合成数据与来自 DeepSeek-V3-Base 的监督数据在写作、事实性问答 (QA) 和自我认知等领域进行合并。

8. SFT involved merging the synthetic data with supervised data from DeepSeek-V3-Base in domains like writing, factual question answering (QA), and self-cognition.

9. 最终强化学习是在各种提示和场景下进行的最终强化学习过程。

10. Final RL was a final reinforcement learning process across diverse prompts and scenarios.

第二步（纯强化学习）是一项重大突破。因为该团队直接将强化学习应用于基础模型（DeepSeek-V3-Base），而没有依赖监督式微调作为预备步骤。这种方法使模型能够探索思维链（CoT）推理来解决复杂问题，最终开发出了DeepSeek-R1-Zero。结果表明，LLM 中的推理能力可以完全通过强化学习来激励，而无需像研究人员长期以来认为的那样进行监督式微调。由于编写人类示例成本高昂，因此使用纯强化学习可以比需要监督式微调的情况下，在更多样化的问题集上进行更长时间的训练。

Step 2 (pure RL) was a major breakthrough because the team directly applied reinforcement learning to the base model (DeepSeek-V3-Base) without relying on supervised fine-tuning as a preliminary step. This approach allowed the model to explore chain-of-thought (CoT) reasoning for solving complex problems, which resulted in the development of DeepSeek-R1-Zero. It turned out that reasoning capabilities in LLMs can be incentivized purely through RL, without the need for SFT, as researchers had long believed. Since human-written examples are expensive, the ability to use pure RL allows for longer training on a much more diverse set of problems than if SFT is required.

为了发挥 DeepSeek-R1 的能力为了提高易用性，该团队创建了模型的精简版本，这些版本可以在配置较低的硬件上运行，同时保留了原模型的大部分推理能力。这些版本包括基于 Qwen 和 Llama 架构的各种规模（15 亿、7 亿、8 亿、14 亿等）的模型。

To make DeepSeek-R1’s capabilities more accessible, the team created distilled versions of the model that can run on more modest hardware while retaining much of the original’s reasoning capability. These include models based on Qwen and Llama architectures in various sizes (1.5B, 7B, 8B, 14B, etc.).

基础模型概览

The Landscape of Foundational Models

GenAI 基础模型生态系统已经发生了显著变化，出现了不同的类别以满足不同的需求。

The GenAI foundational model ecosystem has evolved significantly, with distinct categories emerging to serve different needs.

由于学术基准测试已趋于饱和且容易被操纵，目前公认的评估 GenAI 模型的方法是进行两两盲测。LMArena开展了一项大规模的比较研究，图 1-3显示了 2025 年 4 月的排行榜。

Because academic benchmarks are saturated and can be gamed, the currently most accepted way to rate GenAI models is to compare them pair-wise in blind tests. LMArena carries out a large-scale comparison, and Figure 1-3 shows the April 2025 leaderboard.

图 1-3. 2025 年 4 月 6 日 LMArena 排行榜（带注释的方框由作者添加）

排行榜的纵轴代表车型评分，横轴代表价格。您可以利用此信息确定最符合您预算的车型，以及在特定评分下您应该支付的最低价格。请注意，纵轴上的 Elo 评分与价格呈对数关系（即^2） ，横轴上的价格也是如此。

The leaderboard shows model rating on the y-axis and cost on the x-axis. You can use this to determine the best model that fits your budget and the lowest price you should expect to pay for a given rating. Note that the Elo rating on the y-axis is a logarithmic relationship,² and so is the cost on the x-axis.

在我们截取排行榜截图的当天，综合评分最高的模型（见纵轴）是 Gemini 2.5 Pro Preview、GPT-4.5（预览版）、GPT-4o 和 DeepSeek-R1，Gemini 2.0 Flash 和 OpenAI o1 的排名也接近。随着新模型的发布和更多评分的收集，排名每天都在变化，但排行榜长期以来一直保持着惊人的稳定性，来自 Google、OpenAI 和 Anthropic 的旗舰模型几乎始终名列前茅。这些统称为前沿模型。

The highest-rated models overall (see the y-axis) on the day we screenshotted the leaderboard were Gemini 2.5 Pro Preview, GPT-4.5 (Preview), GPT-4o, and DeepSeek-R1, with Gemini 2.0 Flash and OpenAI o1 also nearly as high. The ranking changes daily as new models are released and more ratings are collected, but the leaderboard has been remarkably consistent over time, with the flagship models from Google, OpenAI, and Anthropic almost always on top. Together, these are referred to as frontier models.

诸如 GPT-5 和 Gemini 2.5 Pro 之类的前沿模型代表了语言模型能力的最高水平，它们在推理、知识和多模态任务方面均表现出色。然而，由于其庞大的规模和专有性质，它们资源消耗巨大且成本高昂，无法在本地运行。它们适用于需要复杂推理且对速度要求不高的企业级应用。前沿模型的最新发展包括多模态功能、增强的推理能力以及扩展的内容窗口（某些模型最多可容纳两百万个词元）。

Frontier models such as GPT 5 and Gemini 2.5 Pro represent the state of the art in language model capabilities, and they offer the highest performance across reasoning, knowledge, and multimodal tasks. However, they are resource intensive and costly and can’t be run locally due to their size and proprietary nature. You’d use them in enterprise-grade applications requiring sophisticated reasoning and where speed is not a concern. Recent developments in frontier models include multimodal capabilities, enhanced reasoning, and extended-content windows (with up to two million tokens in some models).

前沿模型的精简版在性能和效率之间取得平衡，以更低的成本和更快的响应速度提供合理的功能。Gemini Flash、Claude Sonnet 和 GPT-4o-mini 就是很好的例子。这些模型在内容生成和摘要等常见任务上通常表现良好。它们响应速度快，并且对于大批量应用来说性价比很高。图1-3中的 x 轴是对数坐标，Gemini Pro 和 Gemini Flash 之间的成本差异就显而易见了——Flash 的价格比 Gemini Pro 低 20 倍。

Distilled versions of frontier models balance performance with efficiency, offering reasonable capabilities at lower costs and with faster response times. Leading examples include Gemini Flash, Claude Sonnet, and GPT-4o-mini. These models tend to offer good performance on common tasks like content generation and summarization. They also offer fast response times, and they’re cost-effective for high-volume applications. The cost difference between running Gemini Pro and Gemini Flash becomes quite apparent once you note that the x-axis in Figure 1-3 is logarithmic—Flash is 20 times less expensive.

公开组模型有它们的参数公开透明，便于社区改进和自定义。例如 Llama、Mistral、DeepSeek、Qwen 和 Falcon。这些模型性能强劲，但通常落后于前沿模型，它们可以使用专有数据进行微调，但需要更专业的技术来托管。不过，像Together.ai这样的托管 API 服务，以及超大规模数据中心提供的完全托管的 API 端点，都可以解决这个问题。

Open-weight models have their parameters publicly available, allowing for transparency, community improvement, and customization. Examples include Llama, Mistral, DeepSeek, Qwen, and Falcon. These models offer strong performance but generally lag behind frontier models, and they can be fine-tuned on proprietary data but require more expertise to host. However, there are hosted API services such as Together.ai, as well as fully managed API endpoints on the hyperscalers, that address this issue.

本地可托管模型这些产品旨在无需云连接即可在消费级或企业级硬件上运行。例如 Llama 8B 和 Gemma 2B，它们的硬件优化版本可通过NVIDIA NIM获取。这可确保完全的隐私，所有数据都不会离开您的本地设备，也无需支付任何 API 费用。但是，与云模型相比，这些模型的功能显著降低。对于与互联网断开连接的“物理隔离”系统中的前沿模型的需求，一些专有模型供应商也开始提供这项服务——例如，Gemini 可以在 Google 分布式云中本地运行，而OpenAI 可以在 Azure 的本地产品中运行。

Locally hostable models are designed to run on consumer or enterprise hardware without requiring cloud connectivity. Examples include Llama 8B and Gemma 2B, hardware-optimized versions of which are available through NVIDIA NIM. This allows for complete privacy, with no data leaving your local devices and no ongoing API costs. However, these models have significantly reduced capabilities compared to cloud models. The demand for frontier models in “air-gapped” systems that are disconnected from the internet has led some proprietary model vendors to offer this service as well—for example, Gemini can be run on-premises in Google Distributed Cloud and OpenAI can be run on Azure’s on-premises offerings.

自治

Autonomy

在人工智能领域，也需要智能体。具有一定的自主性。在这里，LLM 充当智能体的大脑，因此除了目标（即“及时订购”）之外，你不需要告诉它如何具体管理库存水平。

In AI, an agent is also expected to be somewhat autonomous. Here, the LLM functions as the agent’s brain, so you don’t need to tell it how exactly to manage inventory levels beyond the goal (which is to order “just in time”).

假设你有一份库存商品清单，以及关于这些商品的销售情况和交货时间的数据：

Suppose you have a list of items in your inventory, plus data on how well they’ve been selling and how long they’ll take to deliver:

@dataclass
class InventoryItem:
    name: str
    quantity_on_hand: int
    weekly_quantity_sold_past_n_weeks: [int]
    weeks_to_deliver: int
 
items = [
    InventoryItem("itemA", 300, [50, 70, 80, 100], 2),
    InventoryItem("itemB", 100, [70, 80, 90, 70], 2),
    InventoryItem("itemC", 200, [80, 70, 90, 80], 1)
]

向代理提供物品清单，它会自动计算出需要重新排序的物品：

Provide the list of items to the agent and it will figure out which ones to reorder:

result = agent.run_sync(f"""
Identify which of these items need to be reordered this week.
 
**Items**
{items}
""")

结果将包含以下内容：

The result will include this, in part:

项目B

订购数量=300重新订购原因='当前库存（100）不足以满足预计交货期内的需求。根据最近每周的销售情况（70-90 件），我们需要订购足够的数量以满足两周的交货期需求，并维持安全库存。'

itemB

quantity_to_order=300 reason_to_reorder='Current stock (100) is insufficient to cover projected demand over delivery time. Based on recent weekly sales (70-90 units), we need to order enough to cover the 2-week delivery period plus maintain safety stock.’

相比之下，传统编程需要编写代码来显式管理库存。这种自主性——即无需持续的人工指导或显式编程即可独立运行的能力——是传统软件和人工智能代理之间的关键区别。

Compare this to traditional programming, where you’d have to write code to explicitly manage inventory. Such autonomy—which means the ability to operate independently without constant human guidance or being explicitly programmed to do so—is the key differentiator between traditional software and AI agents.

代理人的特征

Characteristics of Agents

除了自主性之外，智能体通常还应具备以下特征：

Besides autonomy, agents are usually expected to have the following characteristics:

目标导向

Goal orientation

代理程序会朝着特定目标工作，而不是简单地响应输入提示。库存管理代理程序的目标是及时管理库存，该目标已在其系统提示中设定。

Agents work toward specific objectives, rather than simply responding to input prompts. The goal of the inventory manager agent, which is to manage inventory just in time, was set in its system prompt.

计划与推理

Planning and reasoning

请注意，库存管理代理能够规划确定补货数量的步骤。它识别出最近几周的销售范围，预测交货窗口期内的最大销售额，确定所需的商品数量，然后计算出需要补货的商品数量。所有这些步骤都不需要显式编程。

Notice that the inventory manager agent was able to plan the steps to determine how many items to reorder. It identified the range of recent weekly sales, projected the maximum sale forward by the delivery window, determined how many items would be required, and then calculated the number of items to reorder. None of this needed to be explicitly programmed.

感知与行动

Perception and action

智能体可以收集所需数据（“感知”）并根据环境采取行动。通常，您可以通过工具调用（模式 21）赋予智能体这种能力，使其能够调用外部函数（例如搜索网络、调用计算器和写入数据库），我们将在第 7 章中讨论工具调用。借助工具调用，您可以超越仅仅向库存管理员提供商品列表。智能体可以从后端数据库检索库存数量和每周销售数据。此外，它不仅可以告诉您商品 B 需要重新订购，还可以调用 API（甚至是供应商网站上的 API）来下单。

Agents can gather the data they need (“perceive”) and act on their environment. You can usually give them this ability by enabling them to call external functions (such as searching the web, invoking calculators, and writing to databases) through Tool Calling (Pattern 21), which we’ll discuss in Chapter 7. With Tool Calling, you can go beyond explicitly providing the list of items to the inventory manager. Instead, the agent can retrieve the quantity on hand and weekly sales from backend databases. Also, instead of just telling you that item B needs to be reordered, it can invoke an API, perhaps even an API on the vendor’s site, to place the order.

适应能力和学习能力

Adaptability and learning

如何确定订购 300 件商品是正确的？人工库存管理员会根据未来两周的周销售额进行预估，并验证商店不会缺货——智能体也可以做到这一点。在第6章和第9章中，您将看到诸如“反思”（模式 18）和“自我检查”（模式 31）之类的模式，这些模式允许智能体评估其输出并进行自我纠正。

How do you know that ordering 300 items is correct? A human inventory manager would plug in assumptions of weekly sales for the next two weeks and validate that the store will not run out—and an agent can do the same thing. In Chapters 6 and 9 (respectively), you’ll see patterns such as Reflection (Pattern 18) and Self-Check (Pattern 31) that allow an agent to evaluate its output and self-correct.

截至撰写本文时，对于基于基础模型构建的应用而言，实现自主行为仍然是一个值得追求的目标——非确定性、幻觉以及其他各种故障模式都给构建完全自主的人工智能应用带来了挑战。以非确定性为例，每次重新排序时，你得到的列表和项目数量都可能不同。（不妨试试！）规划在简单情况下有效，但在复杂情况下则不然。在第5章中，你会看到诸如“思维链”（模式13）之类的模式，这些模式可以提升智能体进行规划和推理的能力。本书中的许多设计模式旨在使你的人工智能应用更具自主性，或者至少拓展你所能构建的应用的边界。

At the time of writing, agentic behavior remains an aspirational goal for applications built on foundational models—nondeterminism, hallucinations, and various other failure modes pose challenges to building fully autonomous AI applications. Take nondeterminism, for example—each time, you might get a different list and quantity of items to reorder. (Try it!) Planning works in simple cases, but not in hard ones. In Chapter 5, you’ll see patterns such as Chain of Thought (Pattern 13) that improve the ability of an agent to do planning and reasoning. Many of the design patterns in this book are ways to make your AI applications more agentic, or at least to push the boundaries of what you can build.

温度

Temperature

温度（T）是一个该超参数通过在逻辑值传递给函数之前对其进行缩放来控制标记选择的随机性softmax。包含温度的修正softmax方程如下：

Temperature (T) is a hyperparameter that controls the randomness of token selection by scaling the logits before they’re passed through the softmax function. The modified softmax equation with temperature is as follows:

图 1-6显示了用不同的T值缩放相同图表的效果。

The effect of scaling the same graphs with different values of T is shown in Figure 1-6.

如您所见，将温度设置为零会启用贪婪采样。随着温度升高，出现概率较低的“尾部”词被选中的概率也会增加。当分布的初始峰值较低时，温度的影响就不那么明显了。

As you can see, setting the temperature to zero turns on greedy sampling. As the temperature increases, the likelihood that less likely “tail” words will be chosen also increases. The effect of temperature is less pronounced when the distribution was less peaked to begin with.

图 1-6.按温度缩放 logits 的影响

以下是如何调节温度的方法使用 PydanticAI 时：

Here’s how to vary the temperature when using PydanticAI:

agent = Agent('anthropic:claude-3-7-sonnet-latest', 
              model_settings={
                 "temperature": 0.5
              },
              system_prompt="Complete the sentence.")

以下是直接使用 Anthropic API 时的操作方法：

Here’s how to do it when using the Anthropic API directly:

completion = client.messages.create(
    model="claude-3-7-sonnet-latest",
    system="Complete the sentence.",
    temperature=0.5,
    messages=[
        ...
    ]
)

以下是一个例子，展示了“贸易战导致”这一短语在三种不同温度设置下产生的各种后续表达方式：

Here’s an example, showing various continuations of the phrase The trade war caused produced at three different temperature settings:

0.0：贸易战对全球供应链造成了重大干扰，导致消费者面临物价上涨，并给多个行业的企业带来了经济不确定性。

0.5：贸易战对全球供应链造成了严重冲击，导致消费者面临物价上涨，并给多个行业的企业带来经济不确定性。许多制造商被迫重新审视其生产策略，而农民则因报复性关税限制了进入国际市场的机会，面临出口机会减少的困境。其长期影响包括加速供应链多元化，摆脱对受影响地区的依赖，以及重新引发关于贸易保护主义政策有效性的争论。

0.8：贸易战严重扰乱了全球供应链，迫使许多公司重新审视其生产战略和采购政策。这导致进口商品关税上涨、消费者物价上涨，以及受影响行业的经济不确定性。一些企业由于承担额外成本或在海外市场失去市场份额而利润下降。持续的紧张局势也加剧了金融市场的波动，并使相关国家之间的外交关系复杂化。

0.0 : The trade war caused significant disruptions to global supply chains, leading to increased prices for consumers and economic uncertainty for businesses across multiple industries.

0.5 : The trade war caused significant disruptions to global supply chains, leading to increased prices for consumers and economic uncertainty for businesses across multiple industries. Many manufacturers were forced to reconsider their production strategies, while farmers faced reduced export opportunities as retaliatory tariffs limited access to international markets. The long-term effects included accelerated efforts to diversify supply chains away from affected regions and renewed debates about the effectiveness of protectionist trade policies.

0.8 : The trade war caused significant disruptions to global supply chains, forcing many companies to reconsider their manufacturing strategies and sourcing policies. It led to increased tariffs on imported goods, higher prices for consumers, and economic uncertainty in affected industries. Several businesses reported decreased profits as they absorbed additional costs or lost market share in foreign markets. The prolonged tension also contributed to volatility in financial markets and complicated diplomatic relations between the involved nations.

正如你所看到的，提高温度往往会带来更多的创造力。

As you can see, increasing the temperature tends to lead to more creative output.

在第 3 章和第 6 章中，你会看到像 RAG和LLM 作为裁判这样的情况，在这些情况下可能需要使用低温甚至零温度。

In Chapter 3 and Chapter 6, you will see situations, like RAG and LLM-as-Judge, where it may be necessary to use low or even zero temperatures.

Top-K 采样

Top-K Sampling

Top-K采样限制了从词汇表中仅选择最有可能的k 个词元进行标记选择，有效地截断了概率分布的长尾。这有助于避免在高温下出现离谱的后续词形变化。

Top-K sampling restricts token selection to only the k most likely tokens from the vocabulary, effectively truncating the long tail of the probability distribution. This can help you avoid off-the-wall continuations at high temperatures.

以下是设置不同的 top-K 值对延续短语“宇宙飞船”的影响：

Here’s the impact of setting different top-K values to continue the phrase The spaceship:

1：宇宙飞船飞驰在浩瀚的宇宙中，强大的引擎在黑暗中发出蓝光，载着船员们驶向遥远的星辰和未知的冒险。

10：宇宙飞船静静地滑过浩瀚的太空，强大的引擎推动着它驶向遥远的星系，那里以前从未有人类涉足。

100：宇宙飞船翱翔在繁星点点的宇宙中，它闪闪发光的船体反射着遥远外星太阳的光芒，载着船员们驶向已知宇宙边缘之外的未知世界。

1 : The spaceship zoomed through the vast expanse of space, its powerful engines glowing blue against the darkness as it carried its crew toward distant stars and unknown adventures.

10 : The spaceship glided silently through the vast emptiness of space, its powerful engines propelling it toward the distant galaxy where no human had ventured before.

100 : The spaceship soared through the starry expanse, its gleaming hull reflecting the distant light of alien suns as it carried its crew toward unexplored worlds beyond the edge of known space.

如你所见，当 top-K 值较低时，生成的文本与你在现有科幻小说中找到的短语非常相似。

As you can see, when the top-K value is low, the generated text closely follows phrases that you can find in existing science fiction.

细胞核取样

Nucleus Sampling

细胞核采样动态选择最小的词元集合，其累积概率超过阈值p。因此，它也被称为top-P 采样。这创建了一个代表概率质量大部分的词元“核心”。图 1-7展示了对示例分布应用不同的 top-P 值所产生的影响。

Nucleus sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p. Hence, it’s also called top-P sampling. This creates a “nucleus” of tokens that represent the bulk of the probability mass. Figure 1-7 shows the impact of applying varying top-P values to our illustrative distributions.

图 1-7.顶层 P 采样的影响

核心采样会根据模型的置信度在每个步骤进行调整。当模型置信度很高时，考虑的词元较少（类似于低top-K值），而当模型置信度较低时，考虑的词元较多（类似于高top-K值）。因此，与固定top-K值采样相比，核心采样通常能生成更自然的文本。

Nucleus sampling adapts to the model’s confidence at each step. When the model is very confident, few tokens are considered (as with low top-K), and when the model is uncertain, more tokens are considered (as with high top-K). Therefore, it generally produces more natural text than fixed top-K sampling does.

波束搜索

Beam Search

模特并非自行选择。他们并非一次处理一个后续标记，而是采用束搜索，这是一种确定性搜索算法，旨在通过并行探索多个可能的后续标记来找到最可能的标记序列。这样，模型就可以考虑整个序列的概率，而不仅仅是下一个标记的概率。

Models don’t choose continuation tokens one at a time. Instead, they employ beam search, which is a deterministic search algorithm that aims to find the most likely sequence of tokens by exploring multiple possible continuations in parallel. In this way, the model can consider the probability of the entire sequence, not just the next token.

有些型号配有旋钮，可以用来控制波束搜索。

Some models give you knobs to control the beam search.

重复处罚减少已在生成的文本中出现过的词元再次出现的概率增加，从而有助于防止重复循环。这些惩罚措施包括以下几点：

Repetition penalties reduce the probability of tokens that have already appeared in the generated text will appear again, thus helping to prevent repetitive loops. These penalties include the following:

频率惩罚

Frequency penalty

生成的文本中，某个词或短语出现的次数越多，这个数值就越高。

This increases the more a word or phrase appears in the generated text.

出庭处罚

Presence penalty

当单词或短语重复出现时，此规则适用，因此它鼓励模型在生成的文本中使用各种各样的标记。

This applies whenever a word or phrase is repeated, and it thus encourages the model to employ a diverse range of tokens in the generated text.

长度罚分会调整得分根据序列长度进行惩罚。这些惩罚包括以下几种：

Length penalties adjust the scoring of sequences based on their length. These penalties include the following:

最小长度处罚

Minimum length penalty

这迫使模型至少生成一定数量的令牌。

This forces the model to generate at least a certain number of tokens.

最大长度惩罚

Maximum length penalty

这会将生成长度限制在指定范围内。

This caps generation at a specified length.

长度归一化惩罚

Length normalization penalty

该方法将序列得分除以其长度的函数，以防止对较短序列产生偏差。

This divides sequence scores by a function of their length to prevent bias toward shorter sequences.

波束搜索宽度是这个数字在生成过程中会考虑并行序列。您将在第 2 章和第 6 章的模式中看到束搜索的应用。

Beam search width is the number of parallel sequences considered during generation. You will see beam search being used in patterns in Chapter 2 and Chapter 6.

截至撰写本文时，OpenAI 和 Gemini 支持重复惩罚，但 Anthropic 不支持。Hugging Face 中的 Transformers 库支持长度惩罚和束搜索宽度，但托管的模型均不支持。

At the time of writing, OpenAI and Gemini support repetition penalties but Anthropic does not. The Transformers library in Hugging Face supports length penalties and beam search width, but none of the hosted models do.

零样本学习

Zero-Shot Learning

零样本学习涉及指导无需提供任何示例即可执行任务的模型。这项技术依赖于模型的预训练知识及其理解自然语言指令的能力。

Zero-shot learning involves instructing a model to perform a task without providing any examples. This technique relies on the model’s pretrained knowledge and its ability to understand natural language instructions.

例如，您可以要求 Gemini-Flash 执行以下操作：

For example, you could ask Gemini-Flash to do the following:

分析克劳德·莫奈的《日出·印象》中光线的运用，并解释它如何体现印象派的技法。

Analyze the use of light in Claude Monet’s Impression, Sunrise and explain how it exemplifies impressionist techniques.

它会给出类似这样的回复（不过由于模型输出是不确定的，所以你的结果可能会有所不同）：

It would respond with something like this (though your results may vary, since model outputs are nondeterministic):

在《日出·印象》中，莫奈运用光线捕捉转瞬即逝的瞬间，他更注重光线的瞬息效果而非精确的细节。朦胧的氛围和水面上的绚丽倒影，展现了印象派画家对光线感知和主观体验的关注，而非对场景的写实描绘。这种对捕捉光线瞬间“印象”的强调，成为了印象派运动的标志性特征。

In Impression, Sunrise, Monet employs light to capture a fleeting moment, prioritizing its transient effects over precise details. The hazy atmosphere and vibrant reflections on the water showcase the impressionist focus on perception and the subjective experience of light, rather than a realistic depiction of the scene. This emphasis on capturing the immediate “impression” of light became a hallmark of the Impressionist movement.

少样本学习

Few-Shot Learning

小样本学习扩展这种方法通过在提示中包含少量示例来实现。这些示例起到演示作用，帮助模型理解任务的结构和预期输出格式，然后模型会将这种结构应用到新的实例中。由于示例是置于提示的上下文中的，因此这种方法也称为情境学习。

Few-shot learning extends this approach by including a small number of examples in the prompt. The examples serve as demonstrations that help the model understand the task’s structure and expected output format, and the model then applies that structure to new instances. Because the examples are placed in the context of the prompt, this is also called in-context learning.

小样本学习是一种简单的这是一种有效的上下文工程形式，顾名思义，就是在提示的上下文中添加精心设计的数据和指令，以达到预期的效果。本书中，尤其是在第五章，你会遇到许多基于少样本学习的模式。

Few-shot learning is a simple and effective form of context engineering, which is exactly what it sounds like—adding finely crafted data and instructions to the context of the prompt to achieve the desired performance. You’ll encounter many patterns in this book, especially in Chapter 5, that build on few-shot learning.

例如，假设你想开发一个应用程序，允许用户用文字描述他们看到的一幅画作，然后让应用程序识别它。你可以通过提供一个关于克劳德·莫奈的《日出·印象》的示例问答来教它如何识别这幅画：

For example, suppose you want to build an application that allows the user to describe a painting they see in words and have the application identify it. You can teach it what goes into an identification by providing an example question and answer for Impression, Sunrise by Claude Monet:

agent = Agent(MODEL_ID,
              system_prompt="""You are an expert on art history. I will describe
a painting. You should identify it.
""")
 
result = agent.run_sync("""
Example:
```
Description: shows two small rowboats in the foreground and a red Sun.
Answer:
   Painting: Impression, Sunrise
   Artist: Claude Monet
   Year: 1872
   Significance: Gave the Impressionist movement its name; captured the fleeting 
effects of light and atmosphere, with loose brushstrokes.
```
 
Description: The painting shows a group of people eating at a table under an 
outside tent. The men are wearing boating hats.
 
""")

结果正确识别出一幅与描述相符的雷诺阿画作：

The result correctly identifies a Renoir painting that matches the description:

回答：

绘画：船上的午餐聚会

艺术家：皮埃尔-奥古斯特·雷诺阿

年份：1881年

意义：描绘了雷诺阿的朋友们在法国沙图的富尔奈斯餐厅欢聚一堂的场景，展现了美好年代巴黎人无忧无虑的生活氛围；以其鲜艳的色彩、生动的构图以及对光影的描绘而闻名。

Answer:

Painting: Luncheon of the Boating Party

Artist: Pierre-Auguste Renoir

Year: 1881

Significance: Captures a joyful social gathering of Renoir’s friends at the Maison Fournaise restaurant in Chatou, France, depicting the carefree atmosphere of Parisian life during the Belle Époque; known for its vibrant colors, lively composition, and portrayal of light and shadow.

与构建特定领域模型、收集大量训练数据和执行训练任务不同，上下文学习允许逻辑学习模型“即时”执行新任务。示例“引导”逻辑学习模型模仿它们所展示的输入输出模式。

Instead of requiring you to build a domain-specific model, collect lots of training data, and perform the training tasks, in-context learning allows LLMs to perform new tasks “on the fly.” The examples “guide” the LLM to mimic the input-output patterns they demonstrate.

当零样本学习（不提供任何示例）无法解决您的问题时，您可以尝试在提示中添加示例，并为模型提供完成任务的实际指令。使用少量示例进行上下文学习比使用传统机器学习更容易，因为您无需构建庞大的数据集来微调模型。通常，少样本学习也能取得良好的效果。

When zero-shot learning (in which no examples are provided) does not solve your problem, you can try adding examples to your prompt and giving the model practical instructions to complete the task. Using in-context learning with a few examples is easier than using traditional ML because you don’t need to curate an extensive dataset to fine-tune a model. You can often achieve good results with few-shot learning.

借助情境学习，可以快速构建解决方案原型。它还能提升生产应用场景的效率：在提示中更新示例比整理、重新训练和重新部署机器学习模型要快得多。

With in-context learning, it’s possible to quickly prototype solutions. It also boosts production use cases: it’s much faster to update examples in a prompt than to curate, retrain, and redeploy an ML model.

然而，情境式学习也存在一些局限性：

However, in-context learning has a few limitations:

• 只有当基础模型已经具备必要的知识和能力时，这种方法才有效。

• It only works when the foundational model already has the necessary knowledge and capability.

• 添加大量示例会消耗模型上下文窗口中的宝贵标记，从而减慢推理速度。

• Adding many examples consumes valuable tokens of your model’s context window and will slow your inference time.

• 法学硕士有时难以根据几个例子概括更复杂的问题。

• LLMs sometimes struggle to generalize more complex problems based on a few examples.

在这些情况下，培训后干预可能是一种更好的方法。

In these scenarios, post-training might offer a better approach.

培训后方法

Post-Training Methods

回顾图 1-2，训练基础模型包含多个阶段。第一阶段是训练一个基础语言学习模型（LLM），用于预测下一个词；第二阶段是训练基础LLM使用SFT执行任务；第三阶段是进行RLHF训练。你可以使用上述任何一种或全部方法对模型进行后训练。

Recall from Figure 1-2 that training a foundational model involves multiple stages. The first stage involves training a base LLM on predicting the next word, the second stage involves training the base LLM to perform tasks using SFT, and the third stage involves RLHF. You can post-train the model in any or all of these ways.

以下是一些与训练后评估相关的术语，您可能会遇到。这些方法并非完全不同，它们之间存在一些重叠之处：

Here is some terminology you might encounter that is associated with post-training. These are not wholly different methods—there’s some overlap between them:

持续性岗前培训（CPT）

Continued pretraining (CPT)

如果您拥有一个包含基础模型未训练过的词汇（例如行业术语）和关联的数据集，则可以继续训练基础模型。这需要访问基础模型的完整权重和架构。需要注意的是，训练基础模型极其昂贵且耗时，此外，您还需要在这个新模型上执行剩余的训练阶段。2023年3月，彭博社在金融文档上采用了这种方法，几个月后，他们发现基础模型的性能明显优于特定领域的模型。此后，很少有机构选择这种方法。

You can continue training the base model if you have a dataset that contains vocabulary (such as industry jargon) and associations that the foundational model wasn’t trained on. This requires access to the full weights and architecture of the base model. It’s important to note that training base models is extremely expensive and time-consuming, and moreover, you’ll have to perform the remaining stages of training on this new model. In March 2023, Bloomberg used this approach on financial documents, and just months later, it found that foundational models handily outperformed the domain-specific model. Since then, few organizations have chosen this approach.

监督式微调（SFT）

Supervised fine-tuning (SFT)

你可以使用 SFT 以监督的方式，在包含（提示，响应）对的数据集上进一步训练语言模型。如果你使用包含具体指令（例如“改进此管理计划”）的提示以及各种各样的数据，就可以提高模型理解这些特定自然语言指令的能力。如果你用各种各样的指令训练模型，就可以增强其在不同任务上的零样本学习能力。你还可以使用这种指令调优来增强模型的泛化能力，或者引导模型表现出更有益或更诚实的行为。模型是否会表现出这种行为取决于数据集的大小和多样性——如果你只对单个任务（指令）进行 SFT，那么模型很可能会忘记之前学习的任务，并且无法泛化到新任务。另一方面，如果你的指令训练数据集包含多个任务，那么即使只进行一轮 SFT，模型也可能泛化到新任务。

You can use SFT to further train language models on datasets consisting of (prompt, response) pairs in a supervised manner. If you use prompts that consist of specific instructions (“improve this management plan,” etc.) and a wide variety of data, you can improve the model’s ability to follow those specific natural language instructions. If you train the model on a wide variety of instructions, you can enhance its zero-shot capabilities across diverse tasks. You can also use such instruction tuning to enhance generalization or to elicit more helpful or honest behavior from the model. Whether the model will display this behavior will depend on the size and diversity of your dataset—if you do SFT on just a single task (instruction), then the model will probably forget the tasks it learned earlier and not generalize to new tasks. On the other hand, if your instruction training dataset consists of multiple tasks, then the model may generalize to new tasks even after just one round of SFT.

参数高效微调（PeFT）

Parameter-efficient fine-tuning (PeFT)

由于基础模型规模庞大，对其进行进一步训练十分繁琐。因此，参数高效微调（PeFT）方法应运而生，旨在使大型模型的训练过程更加实用。低秩自适应（LoRA）通过低秩分解，使用更小的矩阵来表示权重更新。LoRA 并非微调所有模型参数，而是冻结原始预训练权重，并添加可训练的小型“自适应”矩阵，这些矩阵被分解为低秩表示。LoRA 大幅减少了可训练参数的数量（最多可减少 10,000 倍），并降低了 GPU 内存需求（最多可减少 3 倍）。此外，它不会增加推理延迟，并且通常性能与完全微调相当。量化感知低秩自适应（QLoRA）是 LoRA 的扩展，它量化了模型的所有权重。其训练过程更加节省内存，但速度较慢。另一方面，量化微调模型占用空间更小，因此比 LoRA 速度更快。

Because foundational models are so large, it’s unwieldy to train them further. So, parameter-efficient fine-tuning (PeFT) approaches have emerged to make the training process more practical for large models. Low-rank adaptation (LoRA) represents weight updates using smaller matrices through low-rank decomposition. Instead of fine-tuning all model parameters, LoRA freezes the original pretrained weights and adds small, trainable “adapter” matrices that are decomposed into low-rank representations. LoRA drastically reduces the number of trainable parameters (making them up to 10,000 times fewer) and reduces GPU memory requirements (making them up to 3 times fewer). It also does not add inference latency, and it often performs on par with full fine-tuning. Quantization-aware low-rank adaptation (QLoRA) is an extension of LoRA that quantizes all the weights of the model. Its training process is more memory efficient, albeit slower. On the other hand, the quantized fine-tuned model takes up less space and is therefore faster than LoRA.

偏好调整

Preference tuning

你可以通过让模型针对同一提示生成两个输出，并向其提供关于哪个输出更优的反馈，来进行后训练。当这些偏好由人类提供，且后训练通过强化学习完成时，就称为强化学习反馈（RLHF）。通常，后训练会使用直接偏好优化（DPO）来完成，因为它效率更高。DeepSeek 引入了群体相对策略优化（GRPO），其中生成多个响应，并为每个响应分配一个分数，该分数由群体的平均奖励进行归一化。

You can post-train a model by having it generate two outputs to the same prompt and giving it feedback on which output is better. When such preferences are provided by humans and the post-training is carried out through reinforcement learning, it’s RLHF. Often, the post-training is carried out using direct preference optimization (DPO) because it is more efficient. DeepSeek introduced group relative policy optimization (GRPO), wherein multiple responses are generated and a response is assigned a score that is normalized by the average reward of the group.

你可以进行的后续培训类型是：这与数据集的结构密切相关。如果数据集仅包含文本补全，则可以将其用于无监督训练，例如通过条件补全训练（CPT）教模型学习新词汇或全新的关联。如果数据集包含对输入的理想响应（输入-输出对），则可以将其用于序列优化训练（SFT）或指令调优。如果数据集包含对每个输入的两个响应，并记录了哪个响应更优，则可以将其用于偏好调优。您可以以参数高效的方式或在量化模型上执行任何形式的后训练。

The type of post-training you can do is intimately connected to the structure of the dataset. If the dataset consists purely of text completions, then you can use it to do unsupervised training, such as teaching a model new vocabulary or completely new associations via CPT. If the dataset consists of ideal responses to inputs (input-output pairs), then you can use it for SFT or instruction tuning. If the dataset consists of two responses to each input and notes which of the two is preferable, then you can use it for preference tuning. You can do any of these forms of post-training in a parameter efficient way or on quantized models.

本书中你会遇到几种训练后模式，包括内容优化（第 2 章中的模式 5 ）、适配器调优（第 5 章中的模式 15 ）和提示优化（第 6 章中的模式 20 ）。截至撰写本文时（2025年6月），只有开放权重模型支持上述所有形式的后训练操作。如果您使用的是托管模型，请查阅模型提供商的最新文档，以确定是否支持需要后训练的模式（或者更换为提供所需功能的模型）。

You will encounter post-training in several patterns in this book, including Content Optimization (Pattern 5 in Chapter 2), Adapter Tuning (Pattern 15 in Chapter 5), and Prompt Optimization (Pattern 20 in Chapter 6). At the time of writing (June 2025), open weights models are the only ones that support all the forms of post-training above. If you are using a hosted model, please consult up-to-date documentation from your model provider to determine whether a pattern that requires post-training is possible (or change to a model that provides the needed capability).

微调前沿模型

Fine-Tuning a Frontier Model

像 OpenAI 和 Anthropico 这样的公司除了 AWS 和 Google Cloud 等超大规模云服务提供商之外，其他云服务提供商也利用 SFT 简化了训练后前沿模型的微调流程。用户可以上传包含输入输出对的训练数据集并启动微调过程，最终得到一个经过适配器调优的模型，该模型可以像基础模型一样直接使用。

Companies like OpenAI and Anthropic, as well as hyperscalers such as AWS and Google Cloud, have streamlined the process of post-training frontier models using SFT. It’s possible to upload a training dataset of input-output pairs and launch the fine-tuning process, and the result will be an endpoint of an adapter-tuned model that can be used just like the foundational model.

我们将以 OpenAI 的 GPT 系列模型为例进行说明，但微调 Amazon Bedrock 上的 Anthropic Claude 或在 Vertex AI 上微调 Google 的 Gemini 模型也十分类似。生成训练样本对后（至少需要一百对，但几千对更好），需要将它们存储在 JSON 格式的文件中。然后，您可以按如下方式加载训练数据集：

We’ll illustrate this by using OpenAI’s GPT series of models, but fine-tuning Anthropic Claude on Amazon Bedrock or Google’s Gemini on Vertex AI is quite similar. Once you generate your training pairs (you’ll need at least a hundred pairs, but a couple of thousand pairs are even better), you’ll need to store them in a JSON line-formatted file. You can then load the training dataset as follows:

training_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

数据加载完成后，您可以像这样启动微调作业：

Once the data is loaded, you can kick off a fine-tuning job like this:

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-3.5-turbo"  # Base model to fine-tune
)

微调过程需要几分钟到几个小时不等，具体时间取决于训练数据集和模型的大小。训练完成后，您可以像这样查询微调后的 LLM 模型标识符：

The fine-tuning will take a few minutes to a few hours, depending on the size of the training dataset and the model. Once the training is completed, you can query the model identifier for your fine-tuned LLM like this:

job_status = client.fine_tuning.jobs.retrieve(job.id)
if job_status.status == 'succeeded':
    print(f"Model ID: {job_status.fine_tuned_model}")

结果将是一个如下形式的模型 ID：

The result will be a model ID of the following form:

ft:<基本模型>-0125:<组织名称>::<作业 ID>

在推理 API 中使用此模型 ID 来调用微调后的模型。它的 API 与基础模型相同：

Use this model ID in the inference API to invoke the fine-tuned model. It has the same API as the base model:

completion = client.chat.completions.create(
    model=job_status.fine_tuned_model,  # Use the fine-tuned model
    messages=messages
)
print(completion.choices[0].message.content)

微调开放权重模型

Fine-Tuning an Open-Weight Model

Unsloth.ai为您提供Unsloth 能够对 Gemma 和 Llama 等开放权重 LLM 模型进行微调和训练。您可以在本地硬件上运行 Unsloth，也可以使用其托管的微调服务。

Unsloth.ai provides you with the capability to fine-tune and train open-weight LLMs like Gemma and Llama. You can run Unsloth on your local hardware or use its managed fine-tuning services.

要微调 Llama 3 的 4 位量化版本，首先需要加载模型。及其分词器（该类会将输入文本序列分解成模型期望的标记）：

To fine-tune the 4-bit quantized version of Llama 3, start by loading in the model and its tokenizer (the class that will break an input text sequence into tokens of the sort the model expects):

from unsloth import FastLanguageModel
max_seq_length = 2048
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=True,
    dtype=None,
)

然后，将一组适配器配重连接到 Llama 基本模型上：

Then, attach a set of adapter weights to the base Llama model:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "up_proj", "down_proj", 
                    "o_proj", "gate_proj"], 
    use_gradient_checkpointing="unsloth"
)

在之前的代码中，您指定了LoRA 层的秩（或矩阵大小）以及要应用 LoRA 的模型的具体层。之前的代码还将 LoRA 应用于注意力机制（Q、K 和 V 矩阵）以及各种投影层。

In the previous code, you specified the rank (or matrix size) of the LoRA layer and the specific layers of the model to which they are to be applied. The previous code also applies LoRA to attention mechanisms (Q, K, and V matrices) and various projection layers.

假设您的输入和输出对采用 Unsloth 支持的格式之一，您可以使用以下代码加载数据集：

Assuming that your input and output pairs are in one of the formats Unsloth supports, you can load the dataset using the following code:

dataset = load_dataset("...", split="train")
dataset = dataset.map(apply_template, batched=True)

然后，在一台拥有足够强大 GPU 的机器上，您可以使用以下代码启动训练过程：

Then, on a machine with sufficiently powerful GPUs, you can launch the training process by using the following code:

trainer=SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
)
trainer.train()

模型训练完成后，您可以保存并将其推送到 Hugging Face。在那里，您可以先将基础模型及其适配器层合并成一个模型，然后再推送：

Once the model is trained, you can save it and push it to Hugging Face. There, you can merge the base model and its adapter layer into a single model before pushing it:

model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit")
model.push_to_hub_merged("...", tokenizer, save_method="merged_16bit")

然后，您就可以像使用基础模型一样使用此模型。

Then, you can use this model just like the base model.

考虑因素

Considerations

微调模型可以给你带来LLM 具有独特的功能，可以根据您特定领域的用例进行定制，但精细调整的模型会带来额外的复杂性——因此，请确保精细调整带来的好处值得付出这些额外的努力：

Fine-tuning models gives you the unique ability to customize LLMs to your domain-specific use cases, but fine-tuned models come with additional complexities—so make sure that the benefits of fine-tuning are worth these additional headaches:

数据要求

Data requirements

您需要使用数量更多的样本（超过一百个）来定制 LLM 可以处理的任务范围，而不是仅仅提供几个上下文示例。这意味着您需要在尝试微调 LLM 之前预先收集样本。如果您目前没有时间或资源来做到这一点，可以考虑先进行上下文学习，收集数据，然后再对模型进行微调，以提升其性能并提供更一致、更可靠的输出。

Instead of providing a few in-context examples, you must use a more significant number of samples (more than a hundred) to customize the range of tasks the LLM can handle. This will require you to collect the samples ahead of time before attempting to fine-tune your LLMs. If you don’t have the time or resources right now to do that, consider starting with in-context learning, collecting the data, and then fine-tuning your model later to boost its performance and provide more consistent, reliable outputs.

灾难性遗忘

Catastrophic forgetting

对低阶学习模型 (LLM) 进行微调可能会导致灾难性遗忘，即模型过度依赖微调过程中提供的样本，从而丢失之前习得的知识。这会抹杀 LLM 的主要优势：其全面的世界知识。您可以通过在小型数据集上进行微调，仅进行少量训练轮数，并在微调过程中选择合适的学习率来缓解这个问题。通常，您应该将微调学习率设置为预训练阶段结束时的值（通常在 1e-5 左右）。

Fine-tuning LLMs can lead to catastrophic forgetting, in which the model overemphasizes the examples provided during fine-tuning and loses its previously acquired knowledge. This wipes out LLMs’ primary advantage: their comprehensive world knowledge. You can mitigate this issue by fine-tuning your model on a small dataset, for only a few epochs, and by selecting an appropriate learning rate during the fine-tuning process. Generally, you should begin the fine-tuning learning rate at the value where the pretraining phase concluded (typically around 1e-5).

额外的复杂性

Additional complexity

在将微调后的模型部署到生产环境之前，您需要评估其性能，并确保它没有引入任何不利的结构，例如带有偏见的语言。此外，每当发布基础模型的新版本时，您都必须再次微调模型并重新进行训练。最后，您还需要跟踪用于训练微调模型的训练数据和验证数据的来源。这些都是需要仔细执行的额外任务，而使用上下文学习则要简单得多，因为它只需要您添加少量示例。

Before releasing the fine-tuned model to a production environment, you need to evaluate its performance and make sure it hasn’t picked up any counterproductive constructs, like biased language. Also, whenever a new version of your foundational model is released, you must fine-tune the model again and redo the training. Finally, you need to track the lineage of the training and validation data used to train the fine-tuned model. These are additional tasks that require careful execution, and using in-context learning is far simpler since it only requires you to add a handful of examples.

额外费用

Additional costs

像 OpenAI 这样的服务提供商，对微调模型的推理收取比标准模型更高的单次令牌费用。他们收费更高的原因是，微调模型的指令可以更短，并且能够产生相同甚至更高质量的输出。由于服务提供商根据输入和输出令牌来计算价格，他们会提高令牌价格以弥补托管微调模型的额外成本。

Providers like OpenAI charge a higher per-token rate for inference on fine-tuned models than for requests to their standard models. They charge a higher price because prompts to fine-tune models can be much shorter and can elicit the same or higher-quality output. Since providers calculate prices on input and output tokens, they increase token pricing to recover the overhead costs of hosting your fine-tuned model.

另一方面，当你对开放模型进行微调时，模型推理成本会降低——但你必须支付微调过程中的 GPU 费用。根据基础 LLM 的不同，这笔费用可能从几美元起，但很快就会飙升至每个模型版本数百美元。

On the other hand, when you fine-tune open models, the model inference cost reduces—but you must pay for the GPU costs while performing the fine-tuning procedures. Depending on the base LLM, this can start at a few dollars but quickly spiral to hundreds of dollars per model version.